es_out: support Upstream Servers with configuration overriding #7608

mabrarov · 2023-06-25T08:42:03Z

Implementation of Upstream feature for the Elasticsearch output plugin.

This pull request is based on pull request #1560 and Forward output plugin.

It was tested in a local setup with:

Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.

Debug log is available at flb_es.log.
Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.

Debug log is available at flb_es_upstream.log.

Testing

Example configuration files for the change can be found in mabrarov/elastic-stack repository under fluent-bit-es-cluster directory.
Debug log output from testing the change - see above.
Attached Valgrind output that shows no leaks or memory corruption was found - refer to flb_run_code_analysis.log for the output of command
```
TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_td flb-it-network' ./run_code_analysis.sh
```
[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature - refer to es_out: support Upstream Servers with configuration overriding fluent-bit-docs#1143.

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- Elasticsearch output: HA-aware, multi-node routing with per-node configuration and upstream targeting.
- Formatter testing: per-upstream flush context and flush-context callback support for more accurate plugin tests.
Integrations
- Cloud ID parsing and optional AWS credential/provider flows for Elasticsearch endpoints.
Configuration
- New per-node/upstream options and properties (write operation, upstream selection, per-node overrides).
- Default Elasticsearch port corrected to 9200.
Tests
- Expanded runtime tests for upstream/node overrides, cloud/AWS flows, and flush-context behavior.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

plugins/out_es/es.h

plugins/out_es/es_conf_parse.c

mabrarov · 2023-06-28T09:19:30Z

Hi reviewers,

Is it possible to approve only workflow for this pull request, so that automated checks and build can start?

Thank you.

PettitWesley · 2023-06-28T17:55:39Z

@mabrarov sure

mabrarov · 2023-06-29T04:35:15Z

Hi @PettitWesley,

It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:

flb-rt-in_event_test
flb-rt-out_tcp

I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes.

Help of maintainers is appreciated.

Thank you.

plugins/out_es/es_conf.c

mabrarov · 2023-07-10T10:10:13Z

Hi @PettitWesley,

Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened.

Thank you.

mabrarov · 2023-07-20T15:16:09Z

Hi dear reviewers,

Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review?

Thank you.

mabrarov · 2023-08-22T07:18:01Z

Hi @PettitWesley and @edsiper,

It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have.

Thank you.

coderabbitai

Actionable comments posted: 0

♻️ Duplicate comments (1)

tests/runtime/out_elasticsearch.c (1)
31-37: Still avoid tmpnam(NULL)’s shared static buffer

Calling tmpnam(NULL) hands back a pointer backed by a single static buffer. Even though we duplicate it right away, there’s still a race window (and no thread-safety) if another call to tmpnam happens before the copy completes. The standard way to fix that is to supply your own L_tmpnam-sized buffer (or use mkstemp/tmpnam_s). That keeps each invocation isolated and avoids static-analyzer complaints.
-    upstream_conf_filename = tmpnam(NULL);
-    if (upstream_conf_filename == NULL) {
+    char tmp_name[L_tmpnam];
+
+    if (tmpnam(tmp_name) == NULL) {
         return NULL;
     }
-    upstream_conf_filename = flb_strdup(upstream_conf_filename);
+    upstream_conf_filename = flb_strdup(tmp_name);

🧹 Nitpick comments (2)

plugins/out_es/es_conf_parse.c (1)

111-138: Consider replacing strtok, strcpy, and strcat with safer alternatives.

Lines 111-118 use strtok, and lines 133-138 use strcpy/strcat without explicit bounds checking. While these operate on local buffers with size limits, using safer alternatives would reduce risk:

Replace strtok with strtok_r or manual parsing

Replace strcpy/strcat with strncpy/strncat or snprintf

Per past review comments, this is pre-existing code moved from the old location. However, since this PR introduces new files, addressing these safety concerns would improve the overall codebase quality.

plugins/out_es/es.h (1)

58-172: Ownership flag pattern: functional but contentious.

The flb_elasticsearch_config structure includes numerous own_xxx flags (e.g., own_index, own_type, own_cloud_user, etc.) to track which fields are owned by this config instance vs. shared with other config instances.

Per PR discussion:

Purpose: Prevents double-free when per-upstream-node configs share pointers with the base plugin config

Reviewer concern: This pattern deviates from Fluent Bit coding style; a wrapper struct (e.g., flb_es_sds_t with embedded ownership) was suggested

Author response: Forward plugin doesn't support per-upstream overriding; duplicating all ES options per node is impractical. Open to wrapper approach but notes complications with flb_config_map and offsetof usage.

Current state: The ownership flag pattern is functionally correct—all allocations set flags, and elasticsearch_config_destroy checks flags before freeing. However, the project maintainers should decide whether to accept this pattern or require refactoring to a wrapper-based approach.

This is a design-level decision for maintainers. If the wrapper approach is required, the author will need guidance on integrating custom wrapper types with flb_config_map and offsetof.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 22431f0 and dfdb036.

📒 Files selected for processing (13)

include/fluent-bit/flb_lib.h (2 hunks)
include/fluent-bit/flb_output.h (1 hunks)
plugins/out_es/CMakeLists.txt (1 hunks)
plugins/out_es/es.c (33 hunks)
plugins/out_es/es.h (5 hunks)
plugins/out_es/es_conf.c (1 hunks)
plugins/out_es/es_conf.h (1 hunks)
plugins/out_es/es_conf_parse.c (1 hunks)
plugins/out_es/es_conf_parse.h (1 hunks)
plugins/out_es/es_conf_prop.h (1 hunks)
src/flb_engine_dispatch.c (2 hunks)
src/flb_lib.c (2 hunks)
tests/runtime/out_elasticsearch.c (19 hunks)

✅ Files skipped from review due to trivial changes (1)

plugins/out_es/CMakeLists.txt

🚧 Files skipped from review as they are similar to previous changes (3)

plugins/out_es/es_conf_prop.h
plugins/out_es/es_conf_parse.h
src/flb_engine_dispatch.c

🧰 Additional context used

🧠 Learnings (11)

📚 Learning: 2025-08-29T06:25:27.250Z

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.

Applied to files:

include/fluent-bit/flb_lib.h

📚 Learning: 2025-08-29T06:24:26.170Z

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.

Applied to files:

include/fluent-bit/flb_lib.h

📚 Learning: 2025-08-29T06:24:55.855Z

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.

Applied to files:

include/fluent-bit/flb_lib.h

📚 Learning: 2025-08-29T06:25:27.250Z

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.

Applied to files:

include/fluent-bit/flb_lib.h

📚 Learning: 2025-08-29T06:25:02.561Z

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.

Applied to files:

include/fluent-bit/flb_lib.h

📚 Learning: 2025-08-31T12:46:11.940Z

Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.

Applied to files:

include/fluent-bit/flb_lib.h
plugins/out_es/es.h

📚 Learning: 2025-09-08T11:21:33.975Z

Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.

Applied to files:

plugins/out_es/es_conf_parse.c

📚 Learning: 2025-08-20T02:04:27.220Z

Learnt from: mabrarov
Repo: fluent/fluent-bit PR: 7608
File: plugins/out_es/es.c:648-670
Timestamp: 2025-08-20T02:04:27.220Z
Learning: The author mabrarov prefers function contracts that assume valid parameters for internal functions in the Elasticsearch output plugin, rather than adding defensive null pointer checks, to maintain performance and code clarity.

Applied to files:

plugins/out_es/es_conf_parse.c
plugins/out_es/es.c

📚 Learning: 2025-09-22T15:59:55.794Z

Learnt from: nicknezis
Repo: fluent/fluent-bit PR: 10882
File: plugins/out_http/http.c:112-116
Timestamp: 2025-09-22T15:59:55.794Z
Learning: When users consider bug fixes out of scope for their focused PRs, it's appropriate to create separate GitHub issues to track those concerns rather than expanding the current PR scope.

Applied to files:

plugins/out_es/es.c

📚 Learning: 2025-10-06T17:39:21.606Z

Learnt from: edsiper
Repo: fluent/fluent-bit PR: 10967
File: plugins/in_elasticsearch/in_elasticsearch_bulk_prot.c:266-272
Timestamp: 2025-10-06T17:39:21.606Z
Learning: In the in_elasticsearch plugin, payloads are JSON-based, so tag_key values will always be strings. There's no need to handle binary (FLB_RA_BINARY) values when extracting tags from records in this plugin.

Applied to files:

plugins/out_es/es.c

📚 Learning: 2025-11-11T20:33:42.843Z

Learnt from: mabrarov
Repo: fluent/fluent-bit PR: 7608
File: plugins/out_es/es_conf.c:932-946
Timestamp: 2025-11-11T20:33:42.843Z
Learning: The mk_list_is_empty function in Fluent Bit returns 0 when a list IS empty and -1 when a list is NOT empty. Use `mk_list_is_empty(list) == 0` to check if empty, and `mk_list_is_empty(list) != 0` to check if not empty.

Applied to files:

plugins/out_es/es_conf.c

🧬 Code graph analysis (7)

plugins/out_es/es_conf.h (1)

plugins/out_es/es_conf.c (3)

flb_es_conf_create (865-898)

flb_es_conf_destroy (900-931)

flb_es_upstream_conf (933-947)

include/fluent-bit/flb_lib.h (1)

src/flb_lib.c (1)

flb_output_set_test_flush_ctx_callback (617-649)

plugins/out_es/es_conf_parse.c (7)

src/flb_utils.c (2)

flb_utils_split (464-467)

flb_utils_split_free (477-489)

src/flb_sds.c (2)

flb_sds_create (78-90)

flb_sds_destroy (389-399)

src/flb_slist.c (2)

flb_slist_create (27-31)

flb_slist_add (70-84)

src/tls/flb_tls.c (1)

flb_tls_create (183-232)

include/fluent-bit/flb_mem.h (1)

flb_free (126-128)

src/aws/flb_aws_util.c (1)

flb_aws_client_generator (280-283)

src/aws/flb_aws_credentials.c (1)

flb_standard_chain_provider_create (269-327)

tests/runtime/out_elasticsearch.c (3)

include/fluent-bit/flb_mem.h (1)

flb_free (126-128)

plugins/out_es/es.c (1)

flb_elasticsearch_target (648-670)

src/flb_lib.c (5)

flb_output_set_test_flush_ctx_callback (617-649)

flb_input (266-276)

flb_output (279-289)

flb_output_set (520-551)

flb_output_set_test (584-615)

plugins/out_es/es.c (5)

src/flb_record_accessor.c (2)

flb_ra_get_value_object (803-814)

flb_ra_translate (628-633)

plugins/out_es/es_conf.c (1)

flb_es_upstream_conf (933-947)

src/flb_upstream_ha.c (1)

flb_upstream_ha_node_get (88-110)

src/flb_upstream.c (2)

flb_upstream_conn_get (711-844)

flb_upstream_conn_release (862-947)

src/flb_http_client.c (6)

flb_http_client (814-859)

flb_http_buffer_size (872-883)

flb_http_basic_auth (1323-1327)

flb_http_client_destroy (1688-1695)

flb_http_dummy_client (794-812)

flb_http_do (1572-1632)

plugins/out_es/es_conf.c (5)

src/flb_record_accessor.c (2)

flb_ra_create (271-358)

flb_ra_destroy (232-248)

plugins/out_es/es_conf_parse.c (4)

flb_es_conf_set_cloud_auth (143-193)

flb_es_conf_set_cloud_credentials (37-74)

flb_es_set_aws_unsigned_headers (197-216)

flb_es_conf_set_aws_provider (280-345)

src/flb_output.c (4)

flb_output_get_property (1108-1111)

flb_output_upstream_ha_set (1671-1683)

flb_output_net_default (1071-1081)

flb_output_upstream_set (1578-1669)

src/flb_upstream_ha.c (2)

flb_upstream_ha_from_file (356-514)

flb_upstream_ha_destroy (63-78)

src/flb_upstream.c (2)

flb_upstream_create (290-363)

flb_upstream_destroy (656-698)

plugins/out_es/es.h (1)

plugins/out_es/es.c (1)

flb_elasticsearch_target (648-670)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (81)

GitHub Check: PR - container builds / Windows container images (2025)
GitHub Check: PR - container builds / Windows container images (2022)
GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
GitHub Check: PR - packages build Linux / ubuntu/24.04 package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/8 package build and stage to S3
GitHub Check: PR - packages build Linux / raspbian/bookworm package build and stage to S3
GitHub Check: PR - packages build Linux / ubuntu/24.04.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/trixie.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / ubuntu/22.04.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/trixie package build and stage to S3
GitHub Check: PR - packages build Linux / ubuntu/22.04 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/bullseye.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/bookworm.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/9 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/9.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/buster package build and stage to S3
GitHub Check: PR - packages build Linux / debian/bullseye package build and stage to S3
GitHub Check: PR - packages build Linux / debian/buster.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/9.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/10 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/10 package build and stage to S3
GitHub Check: PR - packages build Linux / debian/bookworm package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/9 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/8 package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/10 package build and stage to S3
GitHub Check: PR - packages build Linux / amazonlinux/2 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/10.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/9.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / almalinux/8.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/8.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / rockylinux/10.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/7.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/10.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/8.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/9 package build and stage to S3
GitHub Check: PR - packages build Linux / amazonlinux/2023.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / amazonlinux/2023 package build and stage to S3
GitHub Check: PR - packages build Linux / amazonlinux/2.arm64v8 package build and stage to S3
GitHub Check: PR - packages build Linux / centos/7 package build and stage to S3
GitHub Check: PR - container builds / arm64/production container image build
GitHub Check: PR - container builds / arm/v7/debug container image build
GitHub Check: PR - container builds / arm/v7/production container image build
GitHub Check: PR - container builds / arm64/debug container image build
GitHub Check: PR - container builds / amd64/debug container image build
GitHub Check: PR - container builds / amd64/production container image build
GitHub Check: PR - packages build MacOS / call-build-macos-package (Apple Silicon macOS runner, macos-14, 3.31.6)
GitHub Check: PR - packages build MacOS / call-build-macos-package (Intel macOS runner, macos-14-large, 3.31.6)
GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_COMPILER_STRICT_POINTER_TYPES=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
GitHub Check: PR - fuzzing test
GitHub Check: pr-compile-centos-7
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-without-cxx (3.31.6)

🔇 Additional comments (14)

plugins/out_es/es_conf.c (7)

41-61: LGTM: Record accessor creation and ownership handling.

The function correctly creates a record accessor for the ID key, sets the ownership flag, and properly resolves the conflict between generate_id and id_key with a warning.

63-98: LGTM: Write operation validation.

The function correctly validates the write operation against allowed values (index, create, update, upsert) and ensures that update/upsert operations have the required ID configuration.

230-238: Verify ownership flag setting for manual index/type allocation.

Lines 231-232 and 236-237 allocate index and type using flb_strdup and set the corresponding own_index and own_type flags. This pattern is correct for managing the lifecycle of these strings.

302-328: Ownership initialization is correct and necessary.

The function correctly copies the base configuration and then resets all ownership flags to FLB_FALSE (lines 304-328). This is essential to prevent double-free when the per-node config shares pointers with the base config. Only when a property is explicitly overridden (and a new allocation is made) does the ownership flag get set to FLB_TRUE.

This pattern addresses the ownership tracking requirement to avoid double-free when pointers are shared between plugin-level config and per-upstream-node overrides.

414-564: Memory management pattern is correct despite partial allocation risk.

Multiple flb_sds_create calls allocate strings for per-node overrides. Each allocation:

Checks for NULL and returns -1 on failure

Sets the corresponding own_xxx flag to FLB_TRUE only after successful allocation

The caller (es_config_ha, line 748) invokes elasticsearch_config_destroy(node_ec) on error, which checks each own_xxx flag before freeing. This ensures that:

Successfully allocated strings are freed

Unallocated or failed allocations are skipped (their flags remain FLB_FALSE)

This pattern correctly handles partial initialization without leaking memory.

594-680: LGTM: Cleanup function correctly respects ownership flags.

The elasticsearch_config_destroy function systematically checks each own_xxx flag before freeing the corresponding resource. This is the correct pattern for managing shared vs. owned data in the per-node configuration model.

768-784: Thorough error cleanup in HA configuration path.

The error handling (lines 768-784) correctly:

Nullifies each upstream node's opaque data

Destroys all previously created node configs from the ctx->configs list

Destroys the HA upstream context

Destroys the main config

This ensures no memory leaks or dangling pointers on failure.

plugins/out_es/es_conf_parse.c (1)

37-74: LGTM: Cloud credentials parsing with correct ownership handling.

The function correctly parses the cloud_auth string and sets ownership flags. If cloud_passwd allocation fails (line 64), the function returns -1 with own_cloud_user = TRUE. The caller will invoke elasticsearch_config_destroy, which checks ownership flags and frees only cloud_user (not the unallocated cloud_passwd), preventing leaks while avoiding double-free.

plugins/out_es/es.c (4)

648-670: LGTM: Target selection logic for HA and non-HA modes.

The flb_elasticsearch_target function correctly:

In non-HA mode: retrieves the single config and sets *node = NULL

In HA mode: selects a node via round-robin and retrieves its associated config

The function contract (per past review discussion) assumes ctx and node parameters are non-NULL, which is appropriate for internal functions where callers are controlled.

912-918: Good addition: NULL check for HTTP client creation.

The NULL check for flb_http_client (lines 914-918) addresses a potential null pointer dereference. This prevents crashes if client creation fails and properly routes to the error cleanup path.

1082-1085: Good addition: NULL check for dummy HTTP client creation.

The NULL check for flb_http_dummy_client (lines 1082-1085) prevents dereferencing a NULL pointer in flb_http_buffer_size and returns a proper error code.

1368-1372: New UPSTREAM configuration property enables HA mode.

The addition of the Upstream configuration property (lines 1368-1372) allows users to specify an upstream configuration file for High Availability mode with multiple Elasticsearch nodes.

plugins/out_es/es.h (2)

27-27: Critical fix: Corrected default port from 92000 to 9200.

The default Elasticsearch port was incorrectly set to 92000. This fix (line 27) corrects it to the standard Elasticsearch port 9200.

188-202: Well-documented public API for target selection.

The flb_elasticsearch_target function is well-documented with clear parameter descriptions and return value semantics for both HA and non-HA modes.

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

…llback contract Signed-off-by: Marat Abrarov <[email protected]>

…with Upstream node configuration For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id. For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*. Signed-off-by: Marat Abrarov <[email protected]>

Signed-off-by: Marat Abrarov <[email protected]>

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

Signed-off-by: Marat Abrarov <[email protected]>

…icate username or password used for AWS authentication Signed-off-by: Marat Abrarov <[email protected]>

…on and retrying buffer flush Signed-off-by: Marat Abrarov <[email protected]>

…ent creation failed Signed-off-by: Marat Abrarov <[email protected]>

mabrarov requested review from PettitWesley, edsiper, fujimotos, koleini, leonardo-albertovich and niedbalski as code owners June 25, 2023 08:42

github-actions bot added the docs-required label Jun 25, 2023

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers with configuration overriding fluent/fluent-bit-docs#1143

Open

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 812020d to ba3382a Compare June 25, 2023 09:06

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers #1560

Open

mabrarov commented Jun 26, 2023

View reviewed changes

plugins/out_es/es.h Show resolved Hide resolved

mabrarov commented Jun 26, 2023

View reviewed changes

plugins/out_es/es_conf_parse.c Show resolved Hide resolved

mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr June 28, 2023 18:22 — with GitHub Actions Inactive

mabrarov commented Jul 7, 2023

View reviewed changes

plugins/out_es/es_conf.c Outdated Show resolved Hide resolved

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from ba3382a to b7cd81b Compare July 8, 2023 10:22

mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr July 10, 2023 12:28 — with GitHub Actions Inactive

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b7cd81b to b81d3f7 Compare July 20, 2023 19:38

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b81d3f7 to f6431c2 Compare September 30, 2023 13:32

mabrarov temporarily deployed to pr November 12, 2025 03:19 — with GitHub Actions Inactive

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 22431f0 to dfdb036 Compare November 12, 2025 11:49

mabrarov temporarily deployed to pr November 12, 2025 11:49 — with GitHub Actions Inactive

coderabbitai bot reviewed Nov 12, 2025

View reviewed changes

mabrarov temporarily deployed to pr November 12, 2025 12:08 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr November 12, 2025 12:09 — with GitHub Actions Inactive

mabrarov added 10 commits January 2, 2026 21:55

pipeline: outputs: es: basic support of Upstream, only parameters kno…

0b7b0be

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>

engine: support of tests which require generation of context passed t…

60e4608

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: fixed tests after modification of formatter ca…

68270f5

…llback contract Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: tests for upstream node configuration properties

350957b

Signed-off-by: Marat Abrarov <[email protected]>

engine: support of tests which require generation of context passed t…

34127ee

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: code deduplication

dfbf716

Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: plugin configuration failure if failed to dupl…

e776bb9

…icate username or password used for AWS authentication Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: fix of memory leak when using AWS authenticati…

0e40530

…on and retrying buffer flush Signed-off-by: Marat Abrarov <[email protected]>

pipeline: outputs: es: avoid zero pointer dereferencing when HTTP cli…

7e6b33b

…ent creation failed Signed-off-by: Marat Abrarov <[email protected]>

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from dfdb036 to 7e6b33b Compare January 3, 2026 13:12

mabrarov temporarily deployed to pr January 3, 2026 13:12 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr January 3, 2026 13:30 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

es_out: support Upstream Servers with configuration overriding #7608

es_out: support Upstream Servers with configuration overriding #7608

Uh oh!

mabrarov commented Jun 25, 2023 •

edited by coderabbitai bot

Loading

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jun 28, 2023

Uh oh!

PettitWesley commented Jun 28, 2023

Uh oh!

mabrarov commented Jun 29, 2023 •

edited

Loading

Uh oh!

Uh oh!

mabrarov commented Jul 10, 2023

Uh oh!

mabrarov commented Jul 20, 2023

Uh oh!

mabrarov commented Aug 22, 2023

Uh oh!

coderabbitai bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

es_out: support Upstream Servers with configuration overriding #7608

Are you sure you want to change the base?

es_out: support Upstream Servers with configuration overriding #7608

Uh oh!

Conversation

mabrarov commented Jun 25, 2023 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jun 28, 2023

Uh oh!

PettitWesley commented Jun 28, 2023

Uh oh!

mabrarov commented Jun 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jul 10, 2023

Uh oh!

mabrarov commented Jul 20, 2023

Uh oh!

mabrarov commented Aug 22, 2023

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mabrarov commented Jun 25, 2023 •

edited by coderabbitai bot

Loading

mabrarov commented Jun 29, 2023 •

edited

Loading